feat: add PULUMI_STACK for compiler run enviroment #578

phramz · 2024-06-03T11:41:56Z

This PR adds a new environment variable PULUMI_STACK to the compiler run.

The possibility to determine the stack from the compiler makes it a lot more flexibel when dealing with stack specific resources.

Here an example (I'm using Jinja2 though the issue itself is not compiler specific):

# Pulumi.yaml
runtime:
  name: yaml
  options:
    compiler: jinja2 --format=yaml Pulumi.yaml values.yaml
    
 resources:
  # stack specific resources for stage
  # {% if 'stage' == 'PULUMI_STACK'|getenv %}{% include 'templates/mailcatcher.yaml.j2' %}{% endif %}
  
  # common resources

A more complex scenario would be to add HA resources to a production stack where an integration system is zonal set up to reduce cloud costs.

My approach seems a bit naive to be honest, but I couldn't figure out a better way without breaking interfaces. Also this solution only works from the Run-call context, so GetRequiredPlugins might not be accurate when plugins are stack specific.

WDYT?

github-actions · 2024-06-03T11:42:27Z

PR is now waiting for a maintainer to take action.

Note for the maintainer: Commands available:

/run-acceptance-tests - used to test run the acceptance tests for the project

tgummerer

Thanks for your contribution! This seems reasonable to me at a glance. Ideally we would also have a test showing that this works.

Also curious if others from @pulumi/platform or @AaronFriel have any opinion on this, or other suggestions of how to implement this.

pkg/pulumiyaml/run.go

pkg/server/server.go

github-actions · 2024-06-03T15:17:57Z

PR is now waiting for a maintainer to take action.

Note for the maintainer: Commands available:

/run-acceptance-tests - used to test run the acceptance tests for the project

phramz · 2024-06-03T15:24:46Z

Ideally we would also have a test showing that this works.

Regarding a test ... I'm not sure where and how to ideally implement such a test. Would be an acceptance test unless we can mock the exec package somehow (probably a little refactoring).

Any thoughts or ideas on this @tgummerer

tgummerer · 2024-06-03T15:30:37Z

I think an integration test would be fine here, maybe creating two different stacks with different outputs based on the environment variables you are introducing here? pkg/tests/integration_test.go has some examples of current integration tests.

CHANGELOG_PENDING.md

github-actions · 2024-06-03T18:39:10Z

PR is now waiting for a maintainer to take action.

Note for the maintainer: Commands available:

/run-acceptance-tests - used to test run the acceptance tests for the project

phramz · 2024-06-03T19:03:33Z

I think an integration test would be fine here, maybe creating two different stacks with different outputs based on the environment variables you are introducing here? pkg/tests/integration_test.go has some examples of current integration tests.

@tgummerer I'm having a hard time setting up the integration tests to run locally. Obviously Pulumi is not using my updated binary as language plugin. So my changes don't have any effect during the test. Is there a doc or could you give me a hint on how to run them locally.

I already pushed an unfinished test PTAL.

Installing the binary will give me an error:

error: the yaml language plugin is bundled with Pulumi, and cannot be directly installed with this command. If you need to reinstall this plugin, reinstall Pulumi via your package manager or install script.

pkg/pulumiyaml/run.go

tgummerer · 2024-06-04T07:29:02Z

/run-acceptance-tests

github-actions · 2024-06-04T07:29:26Z

Please view the results of the PR Build + Acceptance Tests Run Here

tgummerer · 2024-06-04T07:34:21Z

Installing the binary will give me an error:

How exactly were you trying to install the binary? I believe you should be able to run make install to install the binary (and then prepend $GOBIN to the $PATH.) There's also a make test Makefile target that should run all tests. Does that not work for you?

github-actions · 2024-06-04T10:27:01Z

PR is now waiting for a maintainer to take action.

Note for the maintainer: Commands available:

/run-acceptance-tests - used to test run the acceptance tests for the project

phramz · 2024-06-04T10:30:49Z

Installing the binary will give me an error:

How exactly were you trying to install the binary? I believe you should be able to run make install to install the binary (and then prepend $GOBIN to the $PATH.) There's also a make test Makefile target that should run all tests. Does that not work for you?

Thanks! It was indeed a PATH issue having both Pulumi from source and Brew in place.

I finished/fixed the test, fingers crossed it'll pass now.

github-actions · 2024-06-04T10:33:01Z

PR is now waiting for a maintainer to take action.

Note for the maintainer: Commands available:

/run-acceptance-tests - used to test run the acceptance tests for the project

Frassle · 2024-06-04T10:44:44Z

/run-acceptance-tests

github-actions · 2024-06-04T10:45:42Z

Please view the results of the PR Build + Acceptance Tests Run Here

github-actions · 2024-06-04T12:32:31Z

PR is now waiting for a maintainer to take action.

Note for the maintainer: Commands available:

/run-acceptance-tests - used to test run the acceptance tests for the project

tgummerer · 2024-06-04T12:34:02Z

/run-acceptance-tests

github-actions · 2024-06-04T12:36:07Z

Please view the results of the PR Build + Acceptance Tests Run Here

github-actions · 2024-06-04T12:53:16Z

PR is now waiting for a maintainer to take action.

Note for the maintainer: Commands available:

/run-acceptance-tests - used to test run the acceptance tests for the project

tgummerer · 2024-06-04T13:31:18Z

/run-acceptance-tests

github-actions · 2024-06-04T13:32:46Z

Please view the results of the PR Build + Acceptance Tests Run Here

pkg/pulumiyaml/run.go

pkg/server/server.go

github-actions · 2024-06-04T13:58:09Z

PR is now waiting for a maintainer to take action.

Note for the maintainer: Commands available:

/run-acceptance-tests - used to test run the acceptance tests for the project

…d `PULUMI_CONFIG` as environment variable to compiler process

github-actions · 2024-06-04T14:30:28Z

PR is now waiting for a maintainer to take action.

Note for the maintainer: Commands available:

/run-acceptance-tests - used to test run the acceptance tests for the project

tgummerer · 2024-06-04T15:01:19Z

/run-acceptance-tests

github-actions · 2024-06-04T15:01:43Z

Please view the results of the PR Build + Acceptance Tests Run Here

tgummerer

Thank you again, and thanks for the patience while we worked out the best way to also make this backwards compatible.

I'm gonna leave this PR open until tomorrow just in case anyone else has any comments, but this looks good to me!

phramz · 2024-06-04T18:55:02Z

Thank you again, and thanks for the patience while we worked out the best way to also make this backwards compatible.

I'm gonna leave this PR open until tomorrow just in case anyone else has any comments, but this looks good to me!

That went really smooth, kudos and big thanks to you!

phramz requested a review from a team as a code owner June 3, 2024 11:41

tgummerer reviewed Jun 3, 2024

View reviewed changes

pkg/pulumiyaml/run.go Outdated Show resolved Hide resolved

Frassle reviewed Jun 3, 2024

View reviewed changes

pkg/pulumiyaml/run.go Outdated Show resolved Hide resolved

pkg/server/server.go Outdated Show resolved Hide resolved

phramz force-pushed the feat/compiler-env branch from 5bbd98d to 24c73af Compare June 3, 2024 15:17

Frassle reviewed Jun 3, 2024

View reviewed changes

CHANGELOG_PENDING.md Outdated Show resolved Hide resolved

phramz force-pushed the feat/compiler-env branch from 24c73af to 1408c11 Compare June 3, 2024 18:38

tgummerer reviewed Jun 4, 2024

View reviewed changes

pkg/pulumiyaml/run.go Outdated Show resolved Hide resolved

tgummerer mentioned this pull request Jun 4, 2024

Workflow failure: Acceptance Tests (Command) #579

Closed

phramz force-pushed the feat/compiler-env branch from 1408c11 to df11fff Compare June 4, 2024 10:26

phramz force-pushed the feat/compiler-env branch from df11fff to cb98c2f Compare June 4, 2024 10:32

phramz force-pushed the feat/compiler-env branch from cb98c2f to e39a1e6 Compare June 4, 2024 12:31

phramz force-pushed the feat/compiler-env branch from e39a1e6 to 1bc2694 Compare June 4, 2024 12:52

tgummerer reviewed Jun 4, 2024

View reviewed changes

pkg/pulumiyaml/run.go Outdated Show resolved Hide resolved

pkg/server/server.go Outdated Show resolved Hide resolved

phramz force-pushed the feat/compiler-env branch from 1bc2694 to f6622eb Compare June 4, 2024 13:57

feat: Pass PULUMI_STACK, PULUMI_PROJECT, PULUMI_ORGANIZATION an…

316c83a

…d `PULUMI_CONFIG` as environment variable to compiler process

phramz force-pushed the feat/compiler-env branch from f6622eb to 316c83a Compare June 4, 2024 14:30

tgummerer approved these changes Jun 4, 2024

View reviewed changes

tgummerer merged commit aed8f2d into pulumi:main Jun 5, 2024
4 of 5 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add PULUMI_STACK for compiler run enviroment #578

feat: add PULUMI_STACK for compiler run enviroment #578

phramz commented Jun 3, 2024 •

edited

Loading

github-actions bot commented Jun 3, 2024

tgummerer left a comment

github-actions bot commented Jun 3, 2024

phramz commented Jun 3, 2024

tgummerer commented Jun 3, 2024

github-actions bot commented Jun 3, 2024

phramz commented Jun 3, 2024

tgummerer commented Jun 4, 2024

github-actions bot commented Jun 4, 2024

tgummerer commented Jun 4, 2024

github-actions bot commented Jun 4, 2024

phramz commented Jun 4, 2024

github-actions bot commented Jun 4, 2024

Frassle commented Jun 4, 2024

github-actions bot commented Jun 4, 2024

github-actions bot commented Jun 4, 2024

tgummerer commented Jun 4, 2024

github-actions bot commented Jun 4, 2024

github-actions bot commented Jun 4, 2024

tgummerer commented Jun 4, 2024

github-actions bot commented Jun 4, 2024

github-actions bot commented Jun 4, 2024

github-actions bot commented Jun 4, 2024

tgummerer commented Jun 4, 2024

github-actions bot commented Jun 4, 2024

tgummerer left a comment

phramz commented Jun 4, 2024

feat: add PULUMI_STACK for compiler run enviroment #578

feat: add PULUMI_STACK for compiler run enviroment #578

Conversation

phramz commented Jun 3, 2024 • edited Loading

github-actions bot commented Jun 3, 2024

tgummerer left a comment

Choose a reason for hiding this comment

github-actions bot commented Jun 3, 2024

phramz commented Jun 3, 2024

tgummerer commented Jun 3, 2024

github-actions bot commented Jun 3, 2024

phramz commented Jun 3, 2024

tgummerer commented Jun 4, 2024

github-actions bot commented Jun 4, 2024

tgummerer commented Jun 4, 2024

github-actions bot commented Jun 4, 2024

phramz commented Jun 4, 2024

github-actions bot commented Jun 4, 2024

Frassle commented Jun 4, 2024

github-actions bot commented Jun 4, 2024

github-actions bot commented Jun 4, 2024

tgummerer commented Jun 4, 2024

github-actions bot commented Jun 4, 2024

github-actions bot commented Jun 4, 2024

tgummerer commented Jun 4, 2024

github-actions bot commented Jun 4, 2024

github-actions bot commented Jun 4, 2024

github-actions bot commented Jun 4, 2024

tgummerer commented Jun 4, 2024

github-actions bot commented Jun 4, 2024

tgummerer left a comment

Choose a reason for hiding this comment

phramz commented Jun 4, 2024

phramz commented Jun 3, 2024 •

edited

Loading